Touchless button control of surgical instrument using augmented reality

ABSTRACT

A surgical system that includes a surgical instrument, a display device, and a controller. The surgical instrument is configured for insertion at least partially into an internal body cavity of a patient, the surgical instrument having an adjustable setting. The controller is in communication with the surgical instrument and the display device. The controller includes a memory storing the adjustable setting of the surgical instrument and a processor. The processor reads from memory the adjustable setting of the surgical instrument and generates a virtual button corresponding to the adjustable setting of the surgical instrument. The generated virtual button is displayed and selectable by a user to implement a corresponding setting on the surgical instrument.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/867,336, filed Jun. 27, 2019, the entire contents of each of whichare incorporated by reference herein.

FIELD

The present technology is generally related to a surgical system, andmore particularly to a surgical system with touchless button controlusing augmented reality.

BACKGROUND

Many surgical instruments employ adjustable settings that the surgeonmay dynamically change during a procedure. Typically, mechanical buttonsand/or footswitches are utilized to adjust the adjustable settings.Additional buttons, actuators, etc., may be found on a consoleassociated with the surgical instrument for adjusting other adjustablesettings.

SUMMARY

The techniques of this disclosure generally relate to a surgical systemwith touchless button control using augmented reality.

In aspects, the present disclosure provides a surgical system thatincludes a surgical instrument configured for insertion at leastpartially into an internal body cavity of a patient, a display deviceand a controller in communication with the surgical instrument and thedisplay device. The controller includes a memory storing the adjustablesetting of the surgical instrument, and a processor. The processor isconfigured to read from memory the adjustable setting of the surgicalinstrument, generate a virtual button corresponding to the adjustablesetting of the surgical instrument, and display the virtual button onthe display device. The displayed virtual button is selectable by a userto implement a corresponding setting on the surgical instrument.

In aspects, in displaying the virtual button, the display device mayproject the virtual button into the field of view.

In aspects, the display device may be a headset.

In aspects, in displaying the virtual button, the display device mayproject the virtual button onto at least one of: the surgicalinstrument, another surgical instrument, a monitor, or an available freespace within the field of view.

In aspects, the surgical system may further includes an image capturedevice configured to capture movement to determine whether the virtualbutton has been selected.

In aspects, the image capture device may capture movement of a hand or afinger to an area where the virtual button may be projected to determinewhether the virtual button has been selected.

In aspects, the image capture device may capture movement of eyes to anarea where the virtual button may be projected to determine whether thevirtual button has been selected.

In aspects, the surgical instrument may be a tissue resectinginstrument, wherein the tissue resecting instrument may be activatedaccording to the selected virtual button.

In aspects, selecting the virtual button may further cause the processorto generate a virtual adjustment button for the selected virtual buttonto adjust the selected adjustable setting. The processor may display thevirtual adjustment button on the display device selectable by the userto adjust the selected adjustable setting to a desired state and set thesurgical instrument to the desired state.

In aspects, the adjustable setting may include fluid pressure, blademovement, or blade speed.

In aspects, the processor may be further configured to display statusinformation indicating the corresponding setting of the surgicalinstrument.

In another aspect, the disclosure provides a surgical instrument coupledto an image capture device configured to capture movement. In aspects ofthe disclosure the surgical instrument includes a controller incommunication with a display device. The controller includes a memorystoring an adjustable settings of the surgical instrument, and aprocessor. The processor is configured to read from memory theadjustable settings of the surgical instrument, generate a virtualbutton corresponding to the surgical instrument, and display the virtualbutton on the display device. The displayed virtual button is selectableby a user to implement a corresponding setting on the surgicalinstrument. The processor captures movement and generates a virtualadjustment button for the selected virtual button to adjust the selectedadjustable setting. The processor displays the virtual adjustment buttonon the display device selectable by the user to adjust the selectedadjustable setting to a desired state and set the surgical instrument tothe desired state.

In aspects, the display device may be a headset.

In aspects, in displaying the virtual button or the virtual adjustmentbutton, the display device may project the virtual button or the virtualadjustment button onto at least one of: the surgical instrument, anothersurgical instrument, a monitor, or an available free space within thefield of view.

In aspects, the image capture device may capture movement of a hand, afinger, or eyes to an area where the virtual button or the virtualadjustment button may be projected to determine whether the virtualbutton or the virtual adjustment button has been selected.

In aspects, the surgical instrument may be a tissue resectinginstrument, wherein the tissue resecting instrument may be activatedaccording to the selected virtual button or virtual adjustment button toactivate the fluid pressure, blade movement, or blade speed settings.

In aspects, the processor may further configured to display statusinformation indicating the corresponding setting of the surgicalinstrument.

In another aspect, the disclosure provides a surgical system. In aspectof the disclosure the surgical system includes a tissue resectinginstrument having an adjustable setting, an image capture deviceconfigured to capture movement, and a controller in communication with adisplay device. The controller includes a memory storing the adjustablesetting of the tissue resecting instrument, and a processor. Theprocessor is configured to read from memory the adjustable setting ofthe tissue resecting instrument and generate a virtual buttoncorresponding to the adjustable setting of the resecting instrument. Thevirtual button is displayed on the display device selectable by a userto implement a corresponding setting on the tissue resecting instrument.The displayed virtual button is projected onto at least one of: thesurgical instrument, another surgical instrument, a monitor, or anavailable free space within the field of view. The image capture devicecaptures movement of eyes, hands, or fingers to an area where thevirtual button is projected and the processor generates the virtualadjustment button for the selected adjustable setting configured toadjust the selected adjustable setting. The virtual adjustment button isdisplayed on the display device and selectable by the user to adjust theselected adjustable setting to a desired state. The displayed virtualadjustment button is projected onto at least one of: the surgicalinstrument, another surgical instrument, a monitor, or an available freespace within the field of view. The image capture device capturesmovement of a hand, a finger, or eyes to an area where the virtualbutton or the virtual adjustment button is projected to determinewhether the virtual button or the virtual adjustment button has beenselected and sets the tissue resecting instrument to the desired state.

The details of one or more aspects of the disclosure are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages of the techniques described in this disclosurewill be apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of a surgical system incorporating augmentedreality;

FIG. 2 is an exemplary diagram of the surgical system of FIG. 1 in useincluding touchless button; and

FIG. 3 is a perspective view of a surgical instrument of the surgicalsystem of FIG. 1 operably positioned within an endoscope includingtouchless buttons.

DETAILED DESCRIPTION

Particular embodiments of the disclosure are described hereinbelow withreference to the accompanying drawings. In the following description,well-known functions or constructions are not described in detail toavoid obscuring the present disclosure in unnecessary detail.

The following aspects of a surgical system, and in particular, asurgical system with touchless button control using augmented reality,incorporate features to enable access to several adjustable settings andinformation, e.g., of a surgical instrument in use, while maintainingsurgeon interaction with the several adjustable settings without contactor requiring the surgeon to look away from surgical field.

The surgical systems incorporating augmented reality of the disclosuredetailed below may be incorporated into different types of surgicalsystems, instruments, assemblies, or configurations. The particularillustrations and embodiments disclosed herein are merely exemplary anddo not limit the scope or applicability of the disclosed technology.

FIG. 1 illustrates a block diagram of a surgical system 1 and FIG. 2illustrates an exemplary use of the surgical system 1. The surgicalsystem 1 includes a controller 100 that has a processor 102 and a memory104. The surgical system 1 also includes one or more surgicalinstruments 110, a display device 120, and an image capture device 130operably coupled to the controller 100.

FIG. 3 illustrates an exemplary embodiment of the surgical instrument110 of the surgical system 1 operably positioned within another surgicalinstrument, e.g., an endoscope 150. The surgical instrument 110 may be atissue resecting instrument configured for insertion through endoscope150 into an organ (e.g., a uterus, a prostate, a bladder, etc.) of apatient for use in a tissue resecting procedure in the organ. Withrespect to typical surgical instruments, the buttons for operating thesurgical instruments are mechanical and out of reach from the user 3(FIG. 2), such as, for example a footswitch or buttons located on aconsole coupled to the tissue resecting device. Mechanical buttons maybe placed closer to the user 3 (FIG. 2) on the surgical instrument orother adjacent instruments; however, the presence of these buttons maymake it difficult to seal the instruments properly for sterilization anddo not allow for customization.

As noted above, surgical instrument 110 is shown disposed within anendoscope 150. Endoscope 150 includes a housing 152, an elongated sheath154 extending distally from housing 152, one or more valves 156 forfluid input and/or output, a light post 158 extending transversely fromhousing 152, a visualization arm 160 obliquely angled relative tohousing 152 and extending therefrom, and a visualization device 162coupled to visualization arm 160 and extending through elongated sheath154 to a distal end portion thereof.

The surgical instrument 110 includes an end effector assembly 112 and aproximal hub assembly 114. The proximal hub assembly 114 is configuredto connect to a handpiece (not shown) which, in turn, is coupled to acontrol unit (not shown) via a cable to provide power and controlfunctionality to surgical instrument 110, although surgical instrument110 may alternatively or additionally include controls associated withthe handpiece and/or a power source, e.g., battery, disposed withinhandpiece. The handpiece and/or control unit may further include aresector drive, e.g., a motor, configured to cause movement of a cuttingblade 116 at the end effector assembly 112, to cut tissue from theorgan. The surgical instrument 110 is further adapted to connect to afluid management system (not shown) via outflow tubing for applyingsuction to remove fluid, tissue, and debris from a surgical site via thesurgical instrument 110. The fluid management system may additionally oralternatively be coupled to the endoscope 150 to facilitate fluid inflowand/or outflow from the surgical site. The control unit and fluidmanagement system may be integral with one another, coupled to oneanother, or separate from one another.

Turning back to FIG. 2, in conjunction with FIG. 1, illustrated is thesurgical system 1 in use including one or more touchless buttons 305. Indisplaying a touchless button 305, the controller 100 causes theprocessor 102 to read from memory 104 an adjustable setting and/orstatus information relating to the surgical instrument 110, endoscope150 (FIG. 3), the control unit, and/or the fluid management system. Theprocessor 102 then generates one or more virtual operational buttons200, e.g., an on/off button, and/or one or more virtual adjustmentbuttons 205, e.g., increase and decrease buttons, based on the one ormore settings, collectively or individually, of the surgical instrument110, endoscope 150 (FIG. 3), control unit, and/or fluid managementsystem, such as, for example fluid pressure, blade movement, and bladespeed. The processor 102 additionally or alternatively generates one ormore virtual status indicators 210, e.g., numerical displays, charts,graphs, and/or icons, etc., based on the status information,collectively or individually, of the surgical instrument 110, endoscope150 (FIG. 3), control unit, and/or fluid management system, such as, forexample, a current fluid pressure, current blade movement profile, andcurrent blade speed.

The display device 120, in communication with the processor 102 of thecontroller 100, is configured to project the virtual button 200, 205and/or the status indicator 210 corresponding to the adjustable settingand the status information, respectively, of the surgical instrument110, endoscope 150 (FIG. 3), control unit, and/or fluid managementsystem. The display device 120 may be a pair of glasses that projectsthe image onto one of the lenses, such as, for example GOOGLE GLASS®(provided by Google®) both lenses, or on a facial shield, a headset 120a, the surgical instrument 110, the endoscope 150, a surgical drape 8,and/or a monitor 120 b. With additional reference to FIG. 3, the displaydevice 120 (FIG. 1) may project the touchless button 305, e.g., virtualbutton 200, 205 and/or virtual status indicator 210, onto one or moreportions of at least one of the surgical instrument 110 within the fieldof view of the user 3, another surgical instrument 110, e.g., theendoscope 150, within the field of view of the user 3, the monitor 120 bwithin the field of view of the user 3, or any available free spacewithin the field of view of the user 3, such as, for example on asurgical drape 8 on the patient 6. The monitor 120 b may, additionallyor alternatively, overlay the touchless button 305 on an image of thepatient obtained by the image capture device 130. Once the touchlessbutton 305 is projected, the user 3 selects the touchless button 305corresponding to the desired adjustable setting, e.g., to activate thecontrol unit and/or the fluid management system and/or to adjust asetting thereof to set the surgical instrument 110 at a desired state.In various embodiments, instead of being projected, the touchless button305 may be overlaid onto one or more portions of at least one of thesurgical instrument 110, another surgical instrument 110, the monitor,or any available free space within the field of view of the user.

The selection of the touchless button 305 is monitored by the imagecapture device 130. The image capture device 130 captures movement ofthe user 3 during a surgical procedure. In capturing the movement of theuser 3 during the surgical procedure, the image capture device 130captures movement of a hand, a finger or eyes to an area in free spacewhere the touchless button 305 is projected. In embodiments, once atouchless button 305 is selected, e.g., a virtual operational button200, the processor 102 generates or reconfigures the touchless button305, e.g., generating or more virtual adjustment buttons 205 and/or oneor more virtual status indicators 210, corresponding to the selectedvirtual operational button 200. For example, where a virtual operationbutton 200 to activate the blade 116 of the surgical instrument 110 isselected, virtual adjustment buttons 205 to adjust the speed of theblade 116 of the surgical instrument 110 and/or virtual statusindicators 210 indicating the current speed of the blade 116 may begenerated.

The processor 102 in communication with the display device 120, displaysthe virtual adjustment buttons 200, 205 and/or virtual status indicators210 as touchless button 305 on the display device 120. The displaydevice 120 in communications with the controller 100 projects thecorresponding virtual adjustment button(s) 205 and the virtual statusindicator(s) 210. The virtual adjustment button(s) 205 and the virtualstatus indicator(s) 210 may be projected next to the virtual operationalbutton 200 or onto at least one of the surgical instrument 110 withinthe field of view of the user 3, another surgical instrument 110 withinthe field of view of the user 3, the monitor 120 b within the field ofview of the user 3, or any available free space within the field of viewof the user 3.

Alternatively, the controller 100 may read from memory 104 theadjustable setting of the surgical instrument 110 and the processor 102may generate the virtual operational button 200, the virtual adjustmentbutton 205, and the virtual status indicator 210 without prior selectionfrom the user 3. The processor 102 in communication with the displaydevice 120, in such embodiments, displays the virtual operational button200, the virtual adjustment button 205, and the virtual status indicator210 similarly as detailed above.

Upon selection of the virtual operational button 200 and/or the virtualadjustment button 205, the processor 102 transmits a signal to thesurgical instrument 110, the endoscope 150, control unit, and/or thefluid management system to set the surgical instrument 110 to thedesired state. The surgical instrument 110 is activated according to thedesired state received by the processor 102 based on the selection ofthe virtual operational button 200 and/or the virtual adjustment button205. Therefore, for example, the user 3 in a tissue resecting proceduremay utilize the virtual operational button 200 and the virtualadjustment button 205 of the touchless button 305 in lieu of mechanicalbuttons resulting in ease of use, customizable use (wherein the user 3can define the desired positions of the touchless buttons 305 or selectfrom a plurality of layouts of touchless buttons 305), and efficient andproper sealing of the surgical instrument 110 for steam sterilization.Additionally, the user 3 can have access to the virtual operationalbutton 200, the virtual adjustment button 205, and can view statusinformation via the virtual status indicator 210 without looking awayfrom the field of view during the surgical procedure.

It should be understood that various aspects disclosed herein may becombined in different combinations than the combinations specificallypresented in the description and accompanying drawings. It should alsobe understood that, depending on the example, certain acts or events ofany of the processes or methods described herein may be performed in adifferent sequence, may be added, merged, or left out altogether (e.g.,all described acts or events may not be necessary to carry out thetechniques). In addition, while certain aspects of this disclosure aredescribed as being performed by a single module or unit for purposes ofclarity, it should be understood that the techniques of this disclosuremay be performed by a combination of units or modules associated with,for example, a medical device.

In one or more examples, the described techniques may be implemented inhardware, software, firmware, or any combination thereof. If implementedin software, the functions may be stored as one or more instructions orcode on a computer-readable medium and executed by a hardware-basedprocessing unit. Computer-readable media may include non-transitorycomputer-readable media, which corresponds to a tangible medium such asdata storage media (e.g., RAM, ROM, EEPROM, flash memory, or any othermedium that can be used to store desired program code in the form ofinstructions or data structures and that can be accessed by a computer).

Instructions may be executed by one or more processors, such as one ormore digital signal processors (DSPs), general purpose microprocessors,application specific integrated circuits (ASICs), field programmablelogic arrays (FPGAs), or other equivalent integrated or discrete logiccircuitry. Accordingly, the term “processor” as used herein may refer toany of the foregoing structure or any other physical structure suitablefor implementation of the described techniques. Also, the techniquescould be fully implemented in one or more circuits or logic elements.

What is claimed is:
 1. A surgical system, comprising: a surgicalinstrument configured for insertion at least partially into an internalbody cavity of a patient, the surgical instrument having an adjustablesetting; a display device; a controller in communication with thesurgical instrument and the display device, the controller including: amemory storing the adjustable setting of the surgical instrument; and aprocessor configured to: read from memory the adjustable setting of thesurgical instrument; generate a virtual button corresponding to theadjustable setting of the surgical instrument; and display the virtualbutton on the display device, wherein the displayed virtual button isselectable by a user to implement a corresponding setting on thesurgical instrument.
 2. The surgical system of claim 1, wherein indisplaying the virtual button, the display device projects the virtualbutton into the field of view.
 3. The surgical system of claim 2,wherein the display device is a headset.
 4. The surgical system of claim1, wherein in displaying the virtual button, the display device projectsthe virtual button onto at least one of: the surgical instrument,another surgical instrument, a monitor, or an available free spacewithin the field of view.
 5. The surgical system of claim 1, furthercomprising an image capture device configured to capture movement todetermine whether the virtual button has been selected.
 6. The surgicalsystem of claim 5, wherein the image capture device captures movement ofa hand or a finger to an area where the virtual button is projected todetermine whether the virtual button has been selected.
 7. The surgicalsystem of claim 5, wherein the image capture device captures movement ofeyes to an area where the virtual button is projected to determinewhether the virtual button has been selected.
 8. The surgical system ofclaim 1, wherein the surgical instrument is a tissue resectinginstrument, wherein the tissue resecting instrument is activatedaccording to the selected virtual button.
 9. The surgical system ofclaim 1, wherein selecting the virtual button further causes theprocessor to: generate a virtual adjustment button for the selectedvirtual button, wherein the virtual adjustment button is configured toadjust the selected adjustable setting; and display the virtualadjustment button on the display device, wherein the virtual adjustmentbutton is selectable by the user to adjust the selected adjustablesetting to a desired state; set the surgical instrument to the desiredstate.
 10. The surgical system of claim 1, wherein the adjustablesetting includes fluid pressure, blade movement, or blade speed.
 11. Thesurgical system of claim 1, wherein the processor is further configuredto display status information indicating the corresponding setting ofthe surgical instrument.
 12. A surgical instrument coupled to an imagecapture device configured to capture movement, the surgical instrumentcomprising: a controller in communication with a display device, thecontroller including: a memory storing an adjustable settings of thesurgical instrument; and a processor configured to: read from memory theadjustable settings of the surgical instrument; generate a virtualbutton corresponding to the surgical instrument; display the virtualbutton on the display device, wherein the displayed virtual button isselectable by a user to implement a corresponding setting on thesurgical instrument; capture movement and generate a virtual adjustmentbutton for the selected virtual button, wherein the virtual adjustmentbutton is configured to adjust the selected adjustable setting; displaythe virtual adjustment button on the display device, wherein the virtualadjustment button is selectable by the user to adjust the selectedadjustable setting to a desired state; and set the surgical instrumentto the desired state.
 13. The surgical instrument of claim 12, whereinthe display device is a headset.
 14. The surgical instrument of claim12, wherein in displaying the virtual button or the virtual adjustmentbutton, the display device projects the virtual button or the virtualadjustment button onto at least one of: the surgical instrument, anothersurgical instrument, a monitor, or an available free space within thefield of view.
 15. The surgical instrument of claim 12, wherein theimage capture device captures movement of a hand, a finger, or eyes toan area where the virtual button or the virtual adjustment button isprojected to determine whether the virtual button or the virtualadjustment button has been selected.
 16. The surgical instrument ofclaim 12, wherein the surgical instrument is a tissue resectinginstrument, wherein the tissue resecting instrument is activatedaccording to the selected virtual button or virtual adjustment button toactivate the fluid pressure, blade movement, or blade speed settings.17. The surgical instrument of claim 12, wherein the processor isfurther configured to display status information indicating thecorresponding setting of the surgical instrument.
 18. A surgical systemcomprising: a tissue resecting instrument having an adjustable setting;an image capture device configured to capture movement; and a controllerin communication with the tissue resecting instrument and a displaydevice, the controller including: a memory storing the adjustablesetting of the tissue resecting instrument; and a processor configuredto: read from memory the adjustable setting of the tissue resectinginstrument; generate a virtual button corresponding to the adjustablesetting of the resecting instrument; display the virtual button on thedisplay device, wherein the displayed virtual button is selectable by auser to implement a corresponding setting on the tissue resectinginstrument; project the virtual button onto at least one of: thesurgical instrument, another surgical instrument, a monitor, or anavailable free space within the field of view; capture movement of eyes,hands or fingers to an area where the virtual button is projected andgenerate virtual adjustment button for the selected adjustable setting,wherein the virtual adjustment button is configured to adjust theselected adjustable setting; display the virtual adjustment button ofthe selected adjustable setting on the display device, wherein thevirtual adjustment button is selectable by the user to adjust theselected adjustable setting to a desired state; project the virtualadjustment button of the selected adjustable setting onto at least oneof: the surgical instrument, another surgical instrument, a monitor, oran available free space within the field of view; and capture movementof eyes, hands or fingers to an area where the virtual adjustment buttonis projected and set the tissue resecting instrument to the desiredstate.